The Flora Incognita app – Interactive plant species identification

The global loss of biodiversity is among the most urgent environmental problems our time, threatening to compromise stability and functioning ecosystems (Barnosky et al., 2012; Ceballos 2015; Dirzo 2014). Ongoing conservation efforts require an accurate understanding spatiotemporal patterns their change over time (Chapman & Busby, 1994). Deficiencies in both quality availability data currently prevent data-driven decisions, such as land-use planning species assessments (Boakes 2010; Geijzendorffer 2016; Proença 2017). Governments scientific agencies typically lack resources fund long-term by professional scientists therefore recruit volunteers, beginners experts, meet assessment goals (Miller-Rushing 2012). Biodiversity monitoring a labour-intensive task, heavily relying on individual expertise correctly identify field. In-situ identification almost impossible for untrained people challenging even professionals, putting it beyond reach many nature enthusiasts (Bonnet 2018). situation further aggravated increasing shortage skilled taxonomists (Frobel Schlumprecht, 2014; Hopkins Freckleton, 2002). For these reasons, there has long been interest developing automated systems (Gaston O'Neill, 2004; Wäldchen Mäder, 2018b). Initial image-based approaches were proposed 15 years ago but only now have become reliable alternative manual identifications (Affouard 2017; Jones, 2020; Seeland 2019; 2018a; Recent boosts accompanied substantial progress machine learning algorithms, notably deep convolutional neural networks (CNNs; LeCun 2015), pushed ‘production-ready’ state. Automated can significantly contribute research 2020). In this paper, we describe Flora Incognita, mobile application plant observation recording. combines novel advances learning-enabled with field guide. Incognita differs from taking multi-modal interactive approach that not analyses single image depicting unknown incorporates habitat information queries user images one or more complementary organs deliver precise (Rzanny 2021; Wittich first action process choosing which growth form belongs to. We distinguish between forbs, grasses, ferns trees request specific perspectives, example, ‘Take tree's stem’. found requesting detailed users take better pictures. addition automatic classification, also incorporate variables process. parallel querying about plant, automatically transfer occurrence's geolocation (given user's consent) date predict prior later analysis. This prediction botanical, geographical climatic context provides hypothesis are less likely be identified current (Wittich adaptive number succeeding interactions, ask form-specific organs, flower leaf. These pass immediate plausibility check, notifying if unlikely depict part actual before being uploaded classification. terminates when either sufficiently high score no additional gained preceding interactions. run intensive studies determine perspectives characteristic per easy acquire 2017, 2019). used based latest technologies. design continuously retrain three types models repository once significant amount new observations available. First, CNN classifier predicts ranked list candidate depicted image. model uses architecture 88.9 million learnable parameters was itself optimized using (Zoph 2018) trained than 1 cluster GP-GPUs (general-purpose graphics processing units) period several months (full training). Second, feedforward network location embeddings similarity predicting at given presence–absence maps, occurrence records acquired various databases, soil type, land cover, phenological regions. Finally, train recurrent structured learn optimal fusion different sources species. consist multiple feature vectors extracted aforementioned networks. initially focussed native wild growing plants Germany chose widely accepted local vascular taxonomic backbone (Wisskirchen Haeupler, 1998). adopted all included taxa rank except genera Taraxacum, Sorbus, Rubus, Pilosella, Hieracium Oenothera. Species within experts certainty, exceptionally difficult trustworthy training required develop deeper within-app resolution. When evolving taxonomy towards holistic flora covering around world, including those occurring parks gardens, migrated Catalogue Life (CoL) while maintaining non-resolved discussed above. Relying CoL allows us easily include groups. Furthermore, simplifies future exchange platforms like Global Information Facility (GBIF) derive CoL. future, aim achieve greater resolution engaging respective designed system flexible client-server solution consisting scalable micro-services running centre client applications making service accessible usage scenarios. Conceptually, server side consists service, (cp. Figure 1). handles user-generated storing them repository, available across devices providing retraining system. realizes app well batch-wise complete already containing data. services manually reviewed Our multi-platform software ecosystem apps freely Android, iOS Harmony OS, developed open-source web frameworks enable maintain modular codebase ensures maximum reuse consistency functionality applications. adaptively guides desired identification. contrast, Capture (Boho 2020) encourages multi-image upon sync server, digital herbarium. Expert review invited botanists. Additionally, provide programming interface (API) registered external clients other 4,851 launched April 2018 19 languages OS devices. It installed 2.75 times globe. four main scenarios app's home screen 2): through Which requested order determined observation's context, is, season information. requires internet connection transferring metadata receiving results. areas without coverage, device's camera import into process, important pertaining (i.e. location). All analysed cascade 1) computer cluster. will suggest short similar probability. each species, comprehensive fact sheet informative provided. Users confirm correct end commit observation. After confirmed during stored personal list. searched, sorted exported comma-separated values format (CSV). Upon selecting observation, details, revise taxon share via social media. shows identifiable searched name, common genus life combination thereof. its characteristics, ecology, toxicity status. sheets updated meaning they ecological, protection distribution relevant position. contain links national floristic websites in-depth goes average interest. While high-quality supported geographic range, yet. adding content facilitates flexibility. collaborating KOSMOS, publisher renowned analogue guides, integrate customers who purchased ‘Was blüht denn da?’ series (Spohn Future collaborations development children-specific sheets. Where available, select settings. news blog, discuss app-related topics, how pictures suitable keep informed frequently stories topics related diversity. use benchmark holdout dataset non-trained evaluate solution. recent alone identifies taxon-averaged accuracy 83% images. studied overall performance acquiring, where necessary, analysing asked two expert botanists check same 1,000 randomly drawn real-user felt able assess 847 They 93% (787) 7% (60) incorrectly app. majority confused cultivated relatives mixed up living expanding set overcome limitations. Table compares free research-grade apps. document occurrences. Results referenced comparative show achieves state art accuracy. attribute Incognita's varying mainly studies' experimental protocol, solely (Jones, Shapovalov 2020), unavailable wrong preventing analysis identifying non-supported Schmidt Steinecke, However, Jones (2020) still concludes very valuable tool ecologists studies. help detect biological indicators changes spatially temporally great need technology citizens, educational practitioners. semi-automatically captured large spatial scales. Since obtained return interested in, crowdsourced, collected opportunistic potential biases considered focus reflected recorded frequency species’ bias, ubiquitous conspicuous rare cryptic Hence, observed represent broadly distributed, often ruderal nitrophilic At densely populated along roadsides much remote poorly away paths roads. Despite peculiarities, sheer helps appropriately handled Mahecha al. (2021) after accounting prominent biases, vegetation sufficient reconstruct well-known biogeographical patterns. clear indication meaningful indeed support monitoring. One scenario invasive major threat associated economic cost society (estimated nearly 12 billion Euro year Europe; Jeschke Weber Gut, 2004). Invasive conspicuous, attract human approach. 3a illustrates assessing Bunias orientalis, neophyte Germany. spreads rapidly replacing species-rich meadow semi-dry grassland biotopes. Early detection rapid response critical processes spread establishment (European Union, up-to-date high-resolution Nature authorities quickly initiate control measures launch citizen science projects studying questions urban agricultural areas. strength lies producing variety dimension. Another phenology, bioindicator climate change. Barve allowed fine-scale delimitation events specifically purpose. unsystematic assessment, numbers contributing might considerable samples supplement systematic protocols. iterations conducted science, unambiguously having flowering fruiting phases. exact timing record, possible differentiate obtain phenologies 3b reports five fraction total day 2019 2020 correlate expected particular find blooming Galanthus nivalis, Ficaria verna Alliaria petiolata took place earlier 2019, contrary Daucus carota. certain Crataegus monogyna figure, peaks fruiting). With information, deduce distinct phases perform temporal biodiversity. database, expect insights inter-annual variability phenology dynamics future. aspect. From didactic perspective, established means teach diversity schools, universities education institutions. Although automates assist acquiring knowledge answering biodiversity, ecological facts relationships Over teaches manner gestalt identify, do own. student-targeted highlighting species' traits helping students improve skills. Last least, intuitive stimulates awareness stems thousands reviews stores continuous stream feedback channels. Comments ‘At last, we're just “blindly” walking forest!’ ‘This fun way finding out environment’ contribution raising Awareness leads initiatives budgets conservation, United Nations (1992) Convention Biological Diversity (CBD) emphasizes importance public crucial CBD's article 13). convincing, appealing well-implemented inspire previously uninvolved participate initiatives. Foremost, authors thank Capture. Plant helped steadily ways. Further, student helpers multitude volunteering translators make 18 languages. handling editor anonymous reviewers comments greatly improved manuscript. P.M. J.W. involved funding acquisition manuscript writing; D.B., H.C.W., M.S. programming; A.D., M.R., content; P.M., M.R. visualization; H.C.W. provided read approved final peer history https://publons.com/publon/10.1111/2041-210X.13611. stores. An apk file [email protected].